Design and evaluation of mobile computer-assisted pronunciation training tools for second language learning

The quality of speech technology (automatic speech recognition, ASR, and textto- speech, TTS) has considerably improved and, consequently, an increasing number of computer-assisted pronunciation (CAPT) tools has included it. However, pronunciation is one area of teaching that has not been developed enough since there is scarce empirical evidence assessing the effectiveness of tools and games that include speech technology in the field of pronunciation training and teaching. This PhD thesis addresses the design and validation of an innovative CAPT system for smart devices for training second language (L2) pronunciation. Particularly, it aims to improve learner’s L2 pronunciation at the segmental level with a specific set of methodological choices, such as learner’s first and second language connection (L1– L2), minimal pairs, a training cycle of exposure–perception–production, individualistic and social approaches, and the inclusion of ASR and TTS technology. The experimental research conducted applying these methodological choices with real users validates the efficiency of the CAPT prototypes developed for the four main experiments of this dissertation. Data is automatically gathered by the CAPT systems to give an immediate specific feedback to users and to analyze all results. The protocols, metrics, algorithms, and methods necessary to statistically analyze and discuss the results are also detailed. The two main L2 tested during the experimental procedure are American English and Spanish. The different CAPT prototypes designed and validated in this thesis, and the methodological choices that they implement, allow to accurately measuring the relative pronunciation improvement of the individuals who trained with them. Both rater’s subjective scores and CAPT’s objective scores show a strong correlation, being useful in the future to be able to assess a large amount of data and reducing human costs. Results also show an intensive practice supported by a significant number of activities carried out. In the case of the controlled experiments, students who worked with the CAPT tool achieved better pronunciation improvement values than their peers in the traditional in-classroom instruction group. In the case of the challenge-based CAPT learning game proposed, the most active players in the competition kept on playing until the end and achieved significant pronunciation improvement results.Departamento de Informática (Arquitectura y Tecnología de Computadores, Ciencias de la Computación e Inteligencia Artificial, Lenguajes y Sistemas Informáticos)Doctorado en Informátic

Estoñol, a computer-assisted pronunciation training tool for Spanish L1 speakers to improve the pronunciation and perception of Estonian vowels

Over the past few years the number of online language teaching materials for non-native speakers of Estonian has increased. However, they focus mainly on vocabulary and pay little attention to pronunciation. In this study we introduce a computerassisted pronunciation training tool, Estoñol, developed to help native speakers of Spanish to train their perception and production of Estonian vowels. The tool’s training program involves seven vowel contrasts, /i-y/, /u-y/, /ɑ-o/, /ɑ-æ/, /e-æ/, /o-ø/, and /o-ɤ/, which have proven to be difficult for native speakers of Spanish. The training activities include theoretical videos and four training modes (exposure, discrimination, pronunciation, and mixed) in every lesson. The tool is integrated into a pre/post-test design experiment with native speakers of Spanish and Estonian to assess the language learners’ perception and production improvement. It is expected that the tool will have a positive effect on the results, as has been shown in previous studies using similar methodology. Kokkuvõte. Katrin Leppik ja Cristian Tejedor-García: Estoñol, mobiilirakendus hispaania emakeelega eesti keele õppijatele vokaalide häälduse ja taju treenimiseks. Eesti keele õppimiseks on loodud mitmeid e-kursusi ja mobiilirakendusi, kuid need keskenduvad peamiselt sõnavara ja gram matika õpetamisele ning pööravad väga vähe tähelepanu hääldusele. Eesti keele häälduse omandamise lihtsustamiseks töötati välja mobiilirakendus Estoñol, mis on mõeldud hispaania emakeelega eesti keele õppijatele. Varasemad uurimused on näidanud, et hispaania emakeelega eesti keele õppijatele valmistab raskusi vokaalide /ɑ, y, ø, æ, ɤ/ hääldamine. Mobiilirakenduse sisu on jagatud seitsmeks peatükiks, kus on võimalik harjutada vokaalipaaride /i-y/, /u-y/, /ɑ-o/, /ɑ-æ/, /e-æ/, /o-ø/, /o-ɤ/ tajumist ja hääldamist. Iga peatükk algab teoreetilise videoga, millele järgnevad taju- ja hääldusharjutused. Mobiilirakenduse mõju hindamiseks keeleõppija hääldusele ja tajule plaanitakse läbi viia eksperiment. Märksõnad: CAPT, eesti keel, hispaania keel, L2, hääldus, taju, vokaalid, Estoño

TipTopTalk! Aplicación móvil para la mejora de pronunciación multilingüe mediante pares mínimos y gamificación

En la actualidad cada vez es mayor el uso de sistemas automáticos de entrenamiento de la pronunciación o CAPT (Computer-Assisted Pronunciation Teaching) en diversos ámbitos, como la educación o el ocio. Entre las herramientas más utilizadas se encuentran los reconocedores automáticos del habla o ASR (Automatic Speech Recognition) y los sintetizadores de voz o TTS (Text-To-Speech). Las tecnologías mencionadas han mejorado sustancialmente con el paso de los años, presentando, sin embargo, algunas deficiencias que no las hacen perfectas. Además, existen muchas dificultades en el aprendizaje de idiomas que repercuten en desmotivación, abandono o una gran inversión de tiempo y dinero en clases presenciales. Los mencionados obstáculos motivan la elaboración del presente Trabajo Fin de Máster, TipTopTalk! consistente en un juego serio implementado en una aplicación móvil, que pretende la mejora de la pronunciación de la segunda lengua de las personas, permitiendo una realimentación al usuario personalizada e instantánea. Se sigue una estrategia denominada Método de la Cardinalidad Nativa, la cual se divide en tres fases: repetición auditiva de sonidos, sensibilización perceptiva y realización de sonidos. Se fundamenta en una batería de pares mínimos (palabras que solo difieren en un fonema) clasificadas por idioma y dialecto (actualmente: español de España, inglés de USA, chino simplificado, portugués de Portugal y portugués de Brasil). A su vez, se plantea un entorno de gamificación y social gaming que incentiva la auto-motivación y la mejora continua y constante de la pronunciación. También se lleva a cabo un proceso de Learning analytics para recopilar, analizar y presentar datos sobre los usuarios junto a sus interacciones con el sistema, con el objetivo de discernir el método de aprendizaje que se está desarrollando y optimizar la herramienta. Por último, cabe destacar las posibles continuaciones del proyecto como un producto innovador en el mercado de las aplicaciones móviles y/o la investigación en mejora de la realimentación ofrecida al usuario en la mejora de la pronunciación, junto a las herramientas de las Tecnologías de la información y la Comunicación (TIC) empleadas para ello.Departamento de Informática (Arquitectura y Tecnología de Computadores, Ciencias de la Computación e Inteligencia Artificial, Lenguajes y Sistemas Informáticos)Máster en Ingeniería Informátic

Desarrollo de una aplicación para la plataforma social educativa Edmodo

En los tiempos actuales, es innegable que las redes sociales y la enseñanza están llamadas a entenderse, por lo que surgen nuevas plataformas que intentan, precisamente, la integración de las nuevas tecnologías en la educación. Este trabajo pretende analizar, comprender y desarrollar una aplicación web para la plataforma social educativa Edmodo, una emergente plataforma social educativa privada con una gran aceptación, sobre todo en EEUU, y cada vez más en el resto del mundo. La aplicación desarrollada, Class Control consiste en una aplicación web que complementa la funcionalidad de Edmodo, nunca suplantando la ya aportada por la plataforma. Es accesible únicamente por usuarios de Edmodo. Estos usuarios pueden ser profesores, padres y alumnos. Por ello, en la aplicación se diferencian los tres roles de usuario mencionados. Los profesores podrán realizar acciones relacionadas con la asistencia, comportamiento, actitud y evaluación de sus alumnos; mientras que los alumnos y sus padres podrán ver los resultados de dichas acciones. Class Control, cuenta con tecnología Java EE, diseño web adaptable y código robusto y reutilizable para poder seguir ampliando funcionalidades si se quisiera. El método y proceso de software seguido es OpenUP y la notación de los diagramas utilizada es UML2. Por último, en este trabajo se encuentra todo el material necesario para la compresión del mismo, tanto los requisitos, el análisis, el diseño, la implementación y las pruebas realizadas, con los manuales pertinentes y los detalles que se creen necesarios.Grado en Ingeniería Informática de Sistema

Using challenges to enhance a learning game for pronunciation training of English as a second language

Producción CientíficaLearning games have a remarkable potential for education. They provide an emergent form of social participation that deserves the assessment of their usefulness and efficiency in learning processes. This study describes a novel learning game for foreign pronunciation training in which players can challenge each other. Native Spanish speakers performed several pronunciation activities during a one-month competition using a mobile application, designed under a minimal pairs approach, to improve their pronunciation of English as a foreign language. This game took place in a competitive scenario in which students had to challenge other participants in order to get high scores and climb up a leaderboard. Results show intense practice supported by a significant number of activities and playing regularity, so the most active and motivated players in the competition achieved significant pronunciation improvement results. The integration of automatic speech recognition (ASR) and text-to-speech (TTS) technology allowed users to improve their pronunciation while being immersed in a highly motivational game.Ministerio de Economía, Industria y Competitividad - Fondo Europeo de Desarrollo Regional (grant TIN2014-59852-R)Junta de Castilla y Leon (grant VA050G18

ISCA Workshop on Speech and Language Technology in Education (SLATE)

This paper introduces the architecture and interface of a serious game intended for pronunciation training and assessment for Spanish students of English as second language. Users will confront a challenge consisting in the pronunciation of a minimal-pair word battery. Android ASR and TTS tools will prove useful in discerning three different pronunciation proficiency levels, ranging from basic to native. Results also provide evidence of the weaknesses and limitations of present-day technologies. These must be taken into account when defining game dynamics for pedagogical purposes.MEC-FEDER Grant TIN2014-59852-R y la Junta de Castilla y León Regional Grant VA145U1

Towards an Open-Source Dutch Speech Recognition System for the Healthcare Domain

The current largest open-source generic automatic speech recognition (ASR) system for Dutch, Kaldi NL, does not include a domain-specific healthcare jargon in the lexicon. Commercial alternatives (e.g., Google ASR system) are also not suitable for this purpose, not only because of the lexicon issue, but they do not safeguard privacy of sensitive data sufficiently and reliably. These reasons motivate that just a small amount of medical staff employs speech technology in the Netherlands. This paper proposes an innovative ASR training method developed within the Homo Medicinalis (HoMed) project. On the semantic level it specifically targets automatic transcription of doctor-patient consultation recordings with a focus on the use of medicines. In the first stage of HoMed, the Kaldi NL language model (LM) is fine-tuned with lists of Dutch medical terms and transcriptions of Dutch online healthcare news bulletins. Despite the acoustic challenges and linguistic complexity of the domain, we reduced the word error rate (WER) by 5.2%. The proposed method could be employed for ASR domain adaptation to other domains with sensitive and special category data. These promising results allow us to apply this methodology on highly sensitive audiovisual recordings of patient consultations at the Netherlands Institute for Health Services Research (Nivel)

Challenges on the Promising Road to Automatic Speech Recognition of Privacy-Sensitive Dutch Doctor-Patient Consultation Recordings

In this paper we present the currently running PDI-SSH project Homo Medicinalis (HoMed), in which we use machine learning to build an Automatic Speech Recognition (ASR) infrastructure for disclosing privacy-sensitive doctor-patient consultation recordings

ISAPh2018: The 2nd International Symposium on Applied Phonetics

Producción CientíficaThere are many software tools that rely on speech technologies for providing to users L2 pronunciation training in the field of Computer Assisted Pronunciation Training (CAPT) [1]. Currently the most popular mobile and desktop operating systems grant users a free access to several Text-To-Speech (TTS) and Automatic Speech Recognition (ASR) systems. The combination of adequate teaching methods and gamification strategies are expected to increase user engagement, provide an adequate feedback and, at the same time, keep users active and comfortable. This study describes the "Japañol" mobile application, a specific and controlled version of TipTopTalk! , a serious game for anywhere anytime self-learning, especially designed for Japanese learners of Spanish as a foreign language, that allows users to train and to test their pronunciation skills using their own Android mobile phones or Windows PCs.Ministerio de Economía, Industria y Competitividad, FEDER (Project TIN2014- 59852-R

Automatic pronunciation assessment vs. automatic speech recognition: A study of conflicting conditions for L2-English

This study addresses the issue of automatic pronunciation assessment (APA) and its contribution to the teaching of second language (L2) pronunciation. Several attempts have been made at designing such systems, and some have proven operationally successful. However, the automatic assessment of the pronunciation of short words in segmental approaches has still remained a significant challenge. Free and off-the-shelf automatic speech recognition (ASR) systems have been used in integration with other tools with the hopes of facilitating improvement in the domain of computer-assisted pronunciation training (CAPT). The use of ASR in APA stands on the premise that a word that is recognized is intelligible and well-pronounced. Our goal was to explore and test the functionality of Google ASR as the core component within a possible automatic British English pronunciation assessment system. After testing the system against standard and non-standard (foreign) pronunciations provided by participating pronunciation experts as well as non-expert native and non-native speakers of English, we found that Google ASR does not and cannot simultaneously meet two necessary conditions (here defined as intrinsic and derived) for performing as an APA system. Our study concludes with a synthetic view on the requirements of a reliable APA system